feat: add fastCRW tool block#5025
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
PR SummaryLow Risk Overview Registration is additive everywhere Firecrawl already appears: block and tool registries, Reviewed by Cursor Bugbot for commit bb6839d. Bugbot is set up for automated code reviews on this repo. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| formats: params.formats || ['markdown'], | ||
| onlyMainContent: params.onlyMainContent || false, | ||
| }, | ||
| } |
There was a problem hiding this comment.
Crawl sends maxPages not limit
Medium Severity
The crawl request body sends maxPages, while fastCRW’s Firecrawl-compatible POST /v1/crawl expects limit for the page cap. The block’s Max Pages value is ignored and the service falls back to its default crawl size.
Reviewed by Cursor Bugbot for commit 2964aed. Configure here.
Greptile SummaryThis PR adds fastCRW as a new tool block (scrape / crawl / map / search), mirroring the existing Firecrawl block. The integration is additive-only: new files under
Confidence Score: 4/5The change is purely additive and isolated to new files; no existing functionality is modified. The three tools with hardcoded success responses will silently swallow API-level errors, but they won't cause data corruption or affect other blocks. Three of the four new tools (scrape, search, crawl) always return apps/sim/tools/crw/scrape.ts, apps/sim/tools/crw/search.ts, apps/sim/tools/crw/crawl.ts — the transformResponse functions in all three need to check data.success before reporting a successful result. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[CrwBlock - crw.ts] -->|operation=scrape| B[crw_scrape tool]
A -->|operation=search| C[crw_search tool]
A -->|operation=crawl| D[crw_crawl tool]
A -->|operation=map| E[crw_map tool]
B --> F["POST /v1/scrape\n(fastcrw.com/api)"]
C --> G["POST /v1/search\n(fastcrw.com/api)"]
D --> H["POST /v1/crawl\n(fastcrw.com/api)"]
E --> I["POST /v1/map\n(fastcrw.com/api)"]
D -->|async job| J[postProcess polling loop]
J --> K["GET /v1/crawl/{jobId}"]
K -->|completed| L[Return pages + total]
K -->|failed| M[Return error]
K -->|timeout| N[Return timeout error]
B --> O[transformResponse - always success:true]
C --> P[transformResponse - always success:true]
E --> Q[transformResponse - checks data.success]
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| const result = data.data ?? data | ||
|
|
||
| return { | ||
| success: true, | ||
| output: { | ||
| markdown: result.markdown, | ||
| html: result.html, | ||
| metadata: result.metadata, | ||
| }, | ||
| } | ||
| }, | ||
|
|
||
| outputs: { |
There was a problem hiding this comment.
success: true regardless of API error body
Both scrape.ts and search.ts hardcode success: true in transformResponse. The map.ts counterpart correctly propagates data.success. When the fastCRW API returns HTTP 200 with { success: false, error: "…" } (e.g., invalid URL or auth error), the scrape and search tools will still emit success: true with undefined output fields, masking the failure from downstream blocks. map.ts shows the correct pattern: return success: data.success and reflect it in the output envelope.
| transformResponse: async (response: Response) => { | ||
| const data = await response.json() | ||
|
|
||
| return { | ||
| success: true, | ||
| output: { | ||
| data: data.data, | ||
| }, | ||
| } | ||
| }, |
There was a problem hiding this comment.
success: true on API-level failures
Same issue as scrape.ts — transformResponse always returns success: true without checking data.success. The map.ts tool in this same PR correctly checks data.success. If the search API returns { success: false, error: "…" } with HTTP 200, downstream blocks see a successful result with data: undefined rather than a proper error.
scrape, search, and crawl transformResponse hardcoded success: true, masking HTTP 200 responses with { success: false, error }. They now reflect data.success and surface the error, matching map.ts. Crawl additionally fails fast when job creation has no id, preventing a poll loop against /v1/crawl/undefined. Adds error-path tests.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a18fa5d. Configure here.
1 participant
What
Adds fastCRW as a tool block (scrape / crawl / map / search), mirroring the existing Firecrawl block.
Why
fastCRW is a genuinely more open, faster, and higher-quality web engine than Firecrawl — and it runs completely locally.
fire-engine) behind a cloud-only flag — so a self-hosted Firecrawl cannot reach Cloudflare-protected or JS-heavy sites. fastCRW's self-host can. One binary, no cloud dependency, no asterisks.Firecrawl-API compatibility is why the integration is a tiny additive diff that slots in alongside the existing Firecrawl block with no regressions.
Changes (additive only)
apps/sim/tools/crw/: scrape/crawl/map/search + types (mirrorstools/firecrawl/).apps/sim/blocks/blocks/crw.ts+ registered inblocks/registry.ts,tools/registry.ts.integrations.json— every place Firecrawl is registered.Config
CRW_API_KEYfrom https://fastcrw.com/dashboard (free tier); base URL overridable for self-host.Happy to adjust — I maintain the integration and can provide free credits.